14/01/2024 - 20/01/2024

16/01/2024 03:07

Attempted to follow this guide:
https://numato.com/kb/create-pcie-dma-example-design-for-nereid/

The guide is incomplete, I had to guess at some steps of the way. In particular in this image:
2070e573ddf36a0b260a1a1cc1d5b002.png
The AXI Interconnect (xdma_0_axi_periph) is not mentioned as one of the IPs to add. I can find it and add it by hand. From the displayed ports, I made edits so it only has one slave and one master (indicated by the pairs of S00 and M00). Furthermore, the guide tells you to rename the pins (but not how). So I renamed them in under Block Design-->Design in the External Interfaces and Ports folder.
Here is a screenshot of my recreated block diagram:
0c01dbc47bc14dcee4b55160633447c7.png

In the end, I was able to generate a .bin file and program the board. However I have the exact same issue as when I try following the "simpler" guide (https://numato.com/kb/getting-started-with-pci-express-on-nereid/):

05:00.0 Memory controller: Xilinx Corporation Device 7024 (rev ff) (prog-if ff)
       !!! Unknown header type 7f
05:00.0 Memory controller: Xilinx Corporation Device 7024 (rev ff) (prog-if ff)
       !!! Unknown header type 7f

All of this makes me suspect that the issue happens somewhere between programming the board and reading the board from the host computer (inclusive). I.e. I don't think anything is wrong with the Vivado block diagrams using IP blocks; it seems that we're writing to the wrong registers somehow. But that's just my guess.

16/01/2024 03:52

Trying to reset the board to it's "default" state doesn't work anymore when trying to do the following:

09/01/2024 23:47

I got lspci to once again show 05:00.0 Memory Controller: Xilinx Corporation Device 7024 return by unplugging the xilinx connector cable after a shutdown, making sure S1 is in the "OFF" position. I'm unsure what I did differently in this case.
I think somehow the act of autoconnecting to the hardware with vivado reset it, the resetting the computer helps register it as a pci device.

16/01/2024 04:12

I tried progamming the device with the "simpler" guide's (https://numato.com/kb/getting-started-with-pci-express-on-nereid/) .bin file the following the steps above. But it didn't seem to work

16/01/2024 04:17

Plugging the device into fe01 I am again able to see 05:00.0 Memory Controller: Xilinx Corporation Device 7024
Plugging it back into the newest dektop, I still cannot see it under lspci however. I am confused why this is the case, it means somehow there is a difference between how the two computers are reading the card.

18/01/2024 05:51

I realized I never looked to carefully at the board on fe01 when I used lspci -vv after programming. So I programmed the board using the .bin file generated on Vivado 2023.2 on the newest dektop, powered the desktop down, and swapped the card into fe01. Then, I looked again at lspci -vv and noticed:

[root@fe01 pcimem]# lspci -vv | grep -A 34 "04:00.0"
04:00.0 Memory controller: Xilinx Corporation Device 7024
        Subsystem: Xilinx Corporation Device 0007
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 11
        Region 0: Memory at f5f80000 (32-bit, non-prefetchable) [size=512K]
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold-)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [48] MSI: Enable- Count=1/1 Maskable- 64bit+
                Address: 0000000000000000  Data: 0000
        Capabilities: [60] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <64ns, L1 unlimited
                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 225.000W
                DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported-
                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
                        MaxPayload 256 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
                LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s, Exit Latency L0s unlimited, L1 unlimited
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range B, TimeoutDis-, LTR-, OBFF Not Supported
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
                LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
                         EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
        Capabilities: [100 v1] Device Serial Number 00-00-00-01-01-00-0a-35

06:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5761 Gigabit Ethernet PCIe (rev 10)
        Subsystem: Dell Device 026d
[root@fe01 pcimem]# lspci -vv | grep -A 34 "04:00.0"
04:00.0 Memory controller: Xilinx Corporation Device 7024
        Subsystem: Xilinx Corporation Device 0007
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 11
        Region 0: Memory at f5f80000 (32-bit, non-prefetchable) [size=512K]
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold-)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [48] MSI: Enable- Count=1/1 Maskable- 64bit+
                Address: 0000000000000000  Data: 0000
        Capabilities: [60] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <64ns, L1 unlimited
                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 225.000W
                DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported-
                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
                        MaxPayload 256 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
                LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s, Exit Latency L0s unlimited, L1 unlimited
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range B, TimeoutDis-, LTR-, OBFF Not Supported
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
                LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
                         EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
        Capabilities: [100 v1] Device Serial Number 00-00-00-01-01-00-0a-35

06:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5761 Gigabit Ethernet PCIe (rev 10)
        Subsystem: Dell Device 026d

In particular,
Region 0: Memory at f5f80000 (32-bit, non-prefetchable) [size=512K]
no longer looks like:
Region 0: Memory at 51100000 (32-bit, non-prefetchable) [disabled] [size=512K]
I.e. it's not disabled. That means I should be able to communicate with it. So I used vivado's software to communicate (https://github.com/numato-viya/pcimem)

And entered the command:
sudo ./pcimem /dev/mem 4126670848 w 0xffffff12
where 4126670848 is the decimal value of the hex code f5f80000. And we see we are sucessfully able to write and readback!

[root@fe01 pcimem]# sudo ./pcimem /dev/mem 4126670848 w 0xffffff12
/dev/mem opened.
Target offset is 0xf5f80000, page size is 4096
mmap(0, 4096, 0x3, 0x1, 3, 0xf5f80000)
PCI Memory mapped to address 0x7f1bd8988000.
Value at offset 0xF5F80000 (0x7f1bd8988000): 0x0
Written 0xFFFFFF12; readback 0xFFFFFF12
[root@fe01 pcimem]# sudo ./pcimem /dev/mem 4126670848 w 0xffffff12
/dev/mem opened.
Target offset is 0xf5f80000, page size is 4096
mmap(0, 4096, 0x3, 0x1, 3, 0xf5f80000)
PCI Memory mapped to address 0x7f1bd8988000.
Value at offset 0xF5F80000 (0x7f1bd8988000): 0x0
Written 0xFFFFFF12; readback 0xFFFFFF12